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ABSTRACT 

The relationship between several individual 
difference variables and test performance using computerized adaptive 
testing (CAT) was studied with 298 South Korean college students. The 
general individual difference variable selected was test anxiety, and 
the math-related individual difference variables were mathematics 
aptitude, math test anxiety, and math self-concept. Computer-related 
individual difference variables were computer literacy, computer 
anxiety, and computer experience. The CAT was an algebra test. Test 
anxiety and other individual difference variables were measured with 
instruments translated into Korean for the study. The seven 
difference variab]es and their squai"es were submitted to an analysis 
using the RSQUARE Procedure from Statistical Analysis System 
software. Math aptitude, math self-concept, math test anxiety, and 
computer literacy, in that order, were significantly related to test 
performance using CAT, but test anxiety, computer exp£rience, and 
computer anxiety were not. Results suggest that the relationship 
between individual difference variables and test performance using 
CAT may be curvilinear. Two tables present analysis results. 
(Contains 27 references.) (SLD) 
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THE RELATIONSHIPS BETV^-IEEN INDIVIDUAL DIFFERENCE VARIABLES AND 
TEST PERFORMANCE IN COMPUTERIZED ADAPTIVE TESTING 

Research on computerized adaptive testing (CAT) has paid much attention to the efficiency and 
precision of the examinees' ability estimation. However, research on the relationships between examinees' 
demographic and psychological characteristics and CAT has been largely neglected (Legg & Ruhr, 1992; 
Kim, 1993, November). The purpose of this study is to examine the relationships between several 
individual difference variables and test performance using CAT. 

The literafjre review indicated that there are several examinee demographic and psychological 
characteristics that may be related to specific tasks and to computers as a test delivery medium (Kim, 
1993, November). It is essential to consider not only general individual difference variables but also 
specific task-related variables when doing CAT research. In this study, the selected individual difference 
variables include a general individual difference variable (test anxiety) and six specific task-related 
variables (math aptitude, math test anxiety, math self-concept, computer literacy, computer anxiety, and 
computer experience). 

The general individual difference variable selected for this study was test anxiety. Only a few 
studies have investigated the relationships between test anxiety and CAT (Powell, 1991; Rocklin & 
O'DonneU, 1991; Vispoel & Rocklin, 1993). In most previous studies except the PoweU's study (1991), 
test anxiety was negatively correlated with test performance using CAT (Rocklin & O'Donnell, 1991; 
Vispoel & Coffman, 1994; Vispoel & Rocklin, 1993; Vispoel, Rocklin, &. Wang, 1994). 

The selected math-related individual difference variables were math aptitude, math test anxiety, 
and matli self-concept. Although a large volume of literatuxe was available concerning the math-related 
individual difference variables, previously reported research did not examine the relationships between 

math-related variables and CAT scores. 

The selected computer-related individual difference variables were computer literacy, computer 
anxiety, and computer experience. Although computer literacy is an important predictor of computer- 
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related achievement, no previous studies were reported on tiie relationship between computer literacy and 
CAT. Previous research on computerized tests suggests that computer anxiety and computer experience 
may affect test performance. Only a few studies (Legg & Buhr, 1992; Vispoel & Rocklin, 1993) have 
examined the relationships between either computer anxiety or computer experience and CAT 
performance. Some aspects of these variables have been investigated as potentially related to CAT, but 
clear-cut results have not been obtained. 

Considerii:}: the lack of investigation into this area, it seems that the literature strongly supports 
application of CAT to a college classroom testing situation and the expansion of the field of CAT research 
with non-English speaking subjects. If it can be demonstrated that the best variables predict the CAT test 
performance, more educators might make use of CAT in the classroom. In addition, if relationships 
between CAT performance and the selected individual difference variables can be demonstrated, more 
research might explore more specific individual difference variables. 

Method 

Subjects 

The subjects in this study were 208 Korean college students enrolled in educational statistics and 
test and measurement courses for the spring semester of 1994 at the Kyungpook National University, 
Taegu, Korea. There were 149 females and 59 males ranging in age from 19 to 24. 
Instrumentations 

rnmpiitpr Ariaptiv ft AlgPihra Tpgt hf.m Ranlf 

The CAT used a pool of 70 items. These items came from a pool developed to be used for 
identifying introductory statistics students whose basic mathematics skills were in need of remediation 
(Wise, Plake, Johnson, & Roos, 1992). Each item used a four-choice, multiple-choice format. The 
original validation procedure resulted in a final pool of 9? items. Three items were deleted by the original 
developers of the item pool (Wise, Plake, Johnson, & Roos, 1992). After reviewing the 90 algebra test 
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items, the authors of the current study selected 70 items that could be applied to Korean students without 
translation. The adaptive test began with an item of medium difficulty and employed a maximum 
information item selection method thereafter, using the MicroCAT Testing System (Assessment Systems 
Corporation, 1989). The adaptive test terminated when 20 items were administered or when the variance 
of . 10 was reached (Legg & Buhr, 1992). 

Math AptitiiHff Tp.<!t 

The Math Aptitude Test consisted of 20 retired Graduate Record Examination items. Two Korean 
mathematics professors and the first author chose five items in each of four subscales. Each item used a 
five^ption, multiple-choice format. The test were translated into Korean by the first author and reviewed 
by two Korean mathematic professors. They were asked to check the test items and compare the Korean 
version to the English version. The alpha coefficient for the Math Aptitude Test was .74. 

Math T(» qt Anvipty ^Iralft 

The Math Test Anxiety Scale consisted of 10 items from the Mathematics Anxiety Rating Scale 
(Richardson & Suinn, 1972) which were identified by Rounds and Hendel (1980) as loading heavUy on 
mathematics test anxiety. The items were rated on a 5-point Likert-type scale. The alpha coefficient for 
this scale was .88. 

Math Sftlf-r onrftpt Scalft 

The modified 10-item math self-concept scale was used to measure math self-concept in this study. 
Six items from the Self Description Questionnaire (SDQ) HI math subscale (Marsh & O'Neill, 1984) and 
four items firom Math Self-concept Scale (Benson, 1989) were selected based on face validity. The items 
were rated on a 5-point Likert-type scale. The alpha coefficient for this scale was .93. 

f nmpiitpr rnmnp tpnrp Tnctnimftnt 

The Computer Competence Instrument, developed by Martinez and Mead (1988), was used to test 
subjects' computer vocabulary and to measure their knowledge about computers. The instrument consisted 
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of 33 4-choice items. Massoud (1991) reported a rank order correlation between the results from his own 
study and that of Martinez and Mead (1988) as .79 using the Computer Competency Instrument. The 
alpha coefficient for this instrument in the present study was .80. 

rnmpiitpr An^iftty .9ralp. 

The computer anxiety subscale from the Computer Attitude Scale (Loyd Sc. Gressard, 1984) was 
used to measure computer anxiety. The subscale has 10 4-point agree-disagree items. The alpha 
reliability coefficients for the computer anxiety subscale in previous studies were .89 (Gressard & Loyd, 
1986), .86 (Loyd & Gressard, 1984), and .78 (Massoud, 1990). The alpha coefficient in the present smdy 
was .87. 

rnmpiitftr F.ypftripn fp Qiipitinnnairi'. 

The Computer Experience Questionnaire developed by Lee (1986) and modified by Mazzeo, 
Druesne, Raffeld, Checketts, and Muhlstein (1991) was used to measure test-takers' computer experience 
and familiarity. Response options "a" through "e" were recoded as 1 through 5, and scores on the first 
seven items were summed to produce a total score for each individual. Item 8 was not mcluded in the total 
score, since tiie correct ordering of the options (face validity) was not readily apparent. The alpha 
coefficient for this questionnaure was .70. 

Tf-st AnYiftty Tnvpntnry 

The Test Anxiety Inventory (TAI; Spielberger, 1980) was used to measure test anxiety. The TAI 
scale consists of 20 4-point Likert-type items (1 =abnost never, 4=ahnost always). The alpha coefficient 
for this scale was .90. 

Thft Translatinn nf thp; Tngm im^nK into KnrMn 

All of the instnunents except the Math Aptimde Test were translated into Korean using the same 
procedures. The process began with the first author translating the instruments from English into Korean. 
Every attempt was made to provide a Korean version that was a faithful representotion of the English 
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version. The Korean versions of the instruments were be translated back to English by a Korean linguist. 
The back-translated versions were compared with the original English to ensure that the translations were 
accurate. Two Korean educational psychologists and the first author checked the Korean and the back- 
translated English versions of the instruments and compared them to the original English versions. 
Data Collection 

The students completed the seven instruments (Math Aptitude Test, Math Test Anxiety Scale, 
Math Self-Concept Scale, Computer Competence Instrument, Computer Anxiety Scale, Computer 
Experience Questionnaire, and Test Anxiety Inventory) during the first week of class. Instruments were 
assembled into packets and distributed to the students as they arrived for class. Each packet contained the 
seven instruments arranged in random order. 

The computerized adaptive algebra test was then given to each student during the second or third 
week of the semester. When each student arrived for testing at the computer laboratory, he or she was 
administered the CAT on an IBM compatible 486 personal computer. 
Data Analysis 

Means, standard deviations, and alpha coefficients were calculated for each individual difference 
variable. Intercorrelations among the individual difference variables were also computed. The basic 
analysis procedure was the use statistical regression to identify the best combination of the seven difference 
variables for predicting achievement on the CAT. Square of the difference variables were also used in the 
analysis. A correlation matrix was computed for the variables and scattergrams were produced to depict 
the relationship between each difference variable and computer adaptive test performance. 

The seven difference variables and their squares were submitted to an analysis using the 
"RSQUARE PROCEDURE" from the SAS computer software library. The squares of the difference 
variables were included to consider the possibility of curvilinear relationships. The SAS RSQUARE 
procedure runs a regression model with every possible combination of predictor variables (McLean, 
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Lindly. & El-Sayed, 1991, November). The results from this procedure provided multiple correlation 
coefficients (R's) and several diagnostic statistics to assist in choosing the best model. 

Results 

Descriptive data from the seven individual difference variables and computer adaptive test 
performance are presented in Table 1. 



Table I 
Descriptive Data 



1 

Variables 


Mean 


Std Dev 


Ubel 


MATO 


14.59 


3.01 


Math Aptitude 


MSCTO 


24.84 


6.81 


Matli Self-Concept 


MTATO 


35.17 


7.14 


Math Test Anxiety 


CLTO 


23.41 


4.98 


Computer Literacy 


CETO 


12.54 


4.34 


Computer Experience 


CATO 


23.68 


5.66 


Computer Anxiety 


TATO 


38.46 


8.35 


Test Anxiety 


SCORE 


1.79 


0.77 


Score using CAT 



The correlation matrix is shown in Table 2. As can be seen in Tabh 2, the highest bivariate 
correlations between the individual difference variables and computer adaptive test performance us;:ng 
CAT are .63 (math aptitude score), .37 (computer literacy), and .36 (math self-concept). The 
relationships between the anxiety-related variables (test anxiety, math test anxiety, and computer anxiety) 
and computer adaptive test performance were related negatively. 
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Table 2 

Intercorrelations between Individual Difference Variables and Test Performance using CAT 



Variables 


MATO 


MSCTO 


MTATO 


CLTO 






T ATH 
VlWKJ 




MATO 


1.00 


.38 


-.19* 


.45 


. 14 


-.Uj 




(\'\* 
.D J 


MSCTO 




1.00 


-.53* 


.20 


.20 


-. lU 


91* 
-.Zl 




MTATO 










- 18^' 




.51* 


-.15" 


CLTO 








l.OO 


.39* 


-.26* 


-.06 


.37* 


CETO 










l.OO 


-.55-^ 


-.10 


.11 


CATO 












l.OO 


11* 


-.05 


TATO 














l.OO 


-.13 


SCORE 
















l.OO 



f < . 05 

The results showed that math aptitude (.400), computer competence (. 137), math self-concept 
(. 121), math test anxiety (.021), test anxiety (.017), computer experience (.013), and computer anxiety 
(.003) accounted for the largest proportions of computer adaptive test variance when considered 
indiividually (proportions of variance accounted for by each variable are shown in parentheses. 

When the seven individual difference variables were combined into three categories; math-related 
variable (math aptitude and math self-concept), computer-related variable (computer Uteracy and computer 
experience), and anxiety-related variable (test anxiety, math test anxiety, and computer anxiety), the R"s 
were .26, .09, and .02 for the math-related, computer-related, and anxiety-related variables, respectively. 

The largest R's ranged from .4190 with one predictor variable to .4825 with aU 14 predictor 
variables. Considering the maximum R' and the number of variables used, the best model appeared to be 
the one that included MATO (Math aptitude), MCSTO (Math self-concept), MAT02 (Math aytitude 
squared), and MSCT02 (Math self-concept squared). The resulting prediction equation was: CAT Score 
= .5727 - .2326 (MATO) + .1159 (MSCTO) + .0140 (MATO squared) - .0021 (MSCTO squared). The 
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R' associated with these variables was .46, only .02 less than the maximum R's attained using all 14 
predictor variables including the squares. 

DUcussion 

In this study, we examined the relationships between individual difference variables and test 
performance using CAT. We found that math aptitude, math self-concept, math test anxiety, and 
computer competence (literacy) were significantly related to test performance using CAT, while test 
anxiety, computer experience, and computer anxiety were not significantly related. The results showed 
that, in order of magnitude, math aptitude, computer competence, math self-concept, math test anxiety, 
test anxiety, computer experience, and computer anxiety were most related to computer adaptive test 
performance. 

We found that math aptitude had the largest relationship with computerized adaptive test 
performance. Computer competence was more strongly related to than computer experience and computer 
anxiety to test performance using CAT. These findings suggest that cognitive ability is the greatest single 
predictor of student achievement in school (Bloom, 1976). Tlie present findmg that math self-concept was 
significantly related to test performance using CAT is consistent with results ft"om previous CAT research 
(Vispoel, Rocklin, & Wang, 1994). It is also consistent with results from previous PPT (paper-and-pencil 
test) research that found academic self-concept to be positively correlated with measures of academic 
achievement (Hansford & Hattie, 1982; Marsh, 1990, 1992; Marsh & O'NeUl, 1984). 

However, test anxiety and math test anxiety in this study were negatively correlated with test 
performance using CAT, but only math test anxiety \vas significant statistically. These results are not 
consistent with the results other CAT studies (Rocklm & O'Donnell, 1991; Vispoel, Rocklin, & Wang, 
1994) that found test anxiety to be significantly correlated with test performance. Computer experience 
and computer anxiety were not significantly correlated with achievement on computer adaptive tests. 
These results are consistent with the findings from other CAT research (Vispoel, Rocklin, & Wang, 1994; 
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Wise et al., 1989). The absence of significance for computer experience and anxiety effects may have 
occurred because Korean students had experience with computers and were provided sufficient practice 

items before taking the CAT. 

We found that math-related variable was more strongly related to test performance using CAT 
than computer-related and anxiety-related variables. Generally, measures of individual difference are 
strongly related to measures of academic achievement in the same content domain. For example, when 
self-concept and achievement measures relate to the same content domain, the causal Unkages are strongest 
(Marsh, 1990, 1992). Vispoel, Rocldin, and Wang (1994) demonstrated that verbal self-concept was 
significantly correlated with verbal ability estimates in CAT research. Although test anxiety was strongly 
related to test performance for American students, the anxiety-related variable was the least related 
variable to achievement for Korean students. There is at least one possible explanation. Korean students 
come to college with a tremendous amount of test experience from having had to pass the competitive 
college entrance examination. It is possible that they are not only sophisticated in terms of testwiseness, 
but they are not afraid of taking tests of any kind. Perhaps this combination of experience is why they feel 
comfortable taking computerized adaptive tests even though most of them never had taken before. 

The best model to estimate test performance using CAT is the four variable model described in the 
previous section. This result suggests that the relationship between individual difference variables and test 
performance using CAT may be curvilinear. This model should be cross-validated as additional daU are 
collected. 
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